Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce lock contention in WAL #391

Merged
merged 1 commit into from
Sep 19, 2024
Merged

Reduce lock contention in WAL #391

merged 1 commit into from
Sep 19, 2024

Conversation

jwilder
Copy link
Collaborator

@jwilder jwilder commented Sep 19, 2024

In the write path, a Write lock is needed in the slow path to ensure a segment exists. This was getting hit more frquently than expected because we roll segments more quickly and the max segment size triggers a rollover more frequently.

To address, this we create a new segment when we rotate proactive so that the slow patch for writes isn't hit as oftent. The side effect of this is that if writes not frequent, we roll over emtpy segments frequently. If we try to upload them, they show up as ingestion failures in kusto because the file has no rows. To address this issue, we just remove the segment if it only contains the segment magic header bytes.

In the write path, a Write lock is needed in the slow path to ensure
a segment exists.  This was getting hit more frquently than expected
because we roll segments more quickly and the max segment size triggers
a rollover more frequently.

To address, this we create a new segment when we rotate proactive so
that the slow patch for writes isn't hit as oftent.  The side effect
of this is that if writes not frequent, we roll over emtpy segments
frequently.  If we try to upload them, they show up as ingestion failures
in kusto because the file has no rows.  To address this issue, we just
remove the segment if it only contains the segment magic header bytes.
@jwilder jwilder merged commit f13484e into main Sep 19, 2024
4 checks passed
@jwilder jwilder deleted the jwilder/ingestor2 branch September 19, 2024 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants